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Abstract: The geostationary optical remote sensing satellite has the advantages of high temporal 
resolution and wide coverage, which can continuously track and observe ship targets on the sea in 
a large range. However, the ship targets in geostationary satellite remote sensing image are usually 
small and weak, and are easily affected by cloud, island and other factors, which brings great diffi- 
culty to the detection of ship targets. This paper proposes a new method for detecting ships moving 
on the sea surface from geostationary optical remote sensing images: Firstly, the adaptive nonlinear 
gray stretch (ANGS) method is used to enhance the image to highlight the small and weak ship 
targets. Secondly, a multi-scale dual-neighbor difference contrast measure (MDDCM) method is de- 
signed to detect the position of the candidate ship target. Then, the shape characteristics of each 
candidate aera is analyzed to remove false ship targets. Finally, the joint probability data association 
(JPDA) method is used for multi-frame data association and tracking. Experiments show that the 
proposed method can effectively detect and track moving ship targets in GF-4 satellite optical re- 
mote sensing images, and the method has better detection performance compared with other clas- 
sical methods. 


Keywords: geostationary orbit satellites; GF-4 satellites; ship detection; ship tracking; visual sali- 
ency; data association; 


1. Introduction 


Surveillance of sea-surface ships has great significance to the economic development 
of sea areas, marine environmental protection, marine ship management, and fishery 
safety supervision [1]. With the rapid development of aerospace technology, sensor tech- 
nology, computer technology and other technologies, satellite remote sensing technology 
has developed rapidly and has become an important means of maritime ship monitoring 
[2]. For many years, Synthetic Aperture Radar (SAR) images are often used as an im- 
portant means of ship detection and tracking because SAR images are less affected by 
weather and time. Compared with SAR satellite images, optical satellite remote sensing 
images can better reflect the shape of ships and are easy to recognize and interpret man- 
ually. In recent years, high-resolution optical remote sensing images from Low Earth Or- 
bit (LEO) satellites are often used to detect and identify ships on the sea [3-8], because 
high-resolution optical remote sensing images can display rich information about the 
shape and texture of ship targets. However, the disadvantages of small coverage and long 
revisit period of LEO satellites make it unable to meet the requirements of real-time and 
continuous monitoring of ship targets moving on the sea surface. Geostationary Orbit 
(GEO) satellites can continuously observe a large area, and have significant advantages 
such as wide observation range and short observation period [9]. Therefore, the use of 
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GEO satellites for near real-time monitoring and tracking of maritime ships can quickly 
obtain dynamic motion information such as the position, heading, speed, and trajectory 
of moving ships [10]. In 2015, China launched gaofen-4 (GF-4), a medium resolution opti- 
cal remote sensing satellite in geostationary orbit, which has multiple observation modes, 
such as gaze mode and cruise mode, and can perform near real-time observation of ship 
targets moving on the sea [11-14]. 

The extraction of candidate ship targets and the removal of false targets (islands, 
clouds) are two key issues for ship detection in optical images. At present, the mainstream 
target detection methods of optical remote sensing image include: detection method based 
on gray statistical features, detection method based on deep learning and detection 
method based on visual attention mechanism. Methods based on gray statistical features 
[3][14][15] use the feature that the gray value of ships or their wakes is significantly higher 
than that of the sea surface for detection, but this type of method can only be applied to 
the absence of clouds, which is easy to be disturbed by clouds and islands, causing false 
alarms or missed detections; The target detection method based on deep learning 
[16][17][18] extracts the texture and geometric features of the target for detection. How- 
ever, due to the lack of texture information of the target in the GEO optical image, the 
target detection method based on deep learning cannot be applied to medium-resolution 
and low-resolution remote sensing images. The method based on visual attention mecha- 
nism [19][20] can quickly and accurately extract regions of interest from complex scenes, 
and the contrast mechanism, multi-resolution representation, size adaptation and other 
characteristics of human vision system (HVS) make it efficient and robust in small target 
detection. 

Generally, the gray value of ship targets is much lower than that of clouds and islands 
in optical remote sensing images. Due to the long imaging distance, coupled with atmos- 
pheric attenuation and cloud occlusion, it is difficult to see the ship itself in the optical 
remote sensing image of GEO satellite with medium resolution, and it is not easy to detect 
the ship directly. During the movement of the ship, a wake will be formed behind the 
direction of movement, and the spread area of the wake is always larger than the ship 
itself. Moreover, compared with sea surface and hull, ship wake often has stronger optical 
reflection characteristics, which can become the main feature of moving ship for detection. 
Therefore, detecting and analyzing the wake of moving ships is an effective means to de- 
tect moving ships on the sea surface. At present, some researchers have carried out re- 
search on ship target detection and tracking methods based on GEO optical images [15] 
[21] [22] [23], but these methods do not fully consider the wake information of moving 
ships, the ability to resist cloud and island interference is insufficient, which is easy to 
cause false alarms or missed detection. 

The core issue of ship tracking is data association. Classical data association methods 
include Nearest Neighbor (NN) data association methods, various improved methods 
based on NN, and Joint Probability Data Association ([PDA) methods, and Multiple Hy- 
pothesis Tracking (MHT) method. [21] uses the MHT method to achieve the multi-frame 
GF-4 optical satellite remote sensing image ship target associated tracking, while sup- 
pressing false alarms. [22] uses the DCF-CSR algorithm to track ship targets in the remote 
sensing image of geosynchronous orbit optical satellite. Considering that the track cross- 
ing will not occur at the same time for ships moving on the sea, the calculation of MHT 
method is complicated, so JPDA method is more suitable for ship tracking. 

This paper analyzes the wake characteristics of sea moving ships, and proposes a 
moving ship detection and tracking method based on optical remote sensing images of 
GEO satellites. The research object involved is GF-4 satellite panchromatic remote sensing 
images. Firstly, the ANGS method is used to enhance the optical remote sensing image to 
suppress the bright cloud and sea clutter in the image and highlight the weak ship wake. 
Secondly, based on the visual saliency theory, we propose a multi-scale dual-neighbor 
difference contrast measure (MDDCM) method, which calculates the saliency map of the 
image and obtains the location of the salient target; Then, the wake shapes of moving ship 
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targets are analyzed, and the false targets such as speckle clouds are removed from the 100 
candidate targets by the shape verification of ship wakes; Finally, JPDA method is used 101 
to carry out data association and multi-target tracking for multi-frame images to obtain 102 
the speed and track of real moving ship targets, and further remove the false targets such 103 
as stationary islands. 104 

The following chapters are arranged as follows: In Section 2, we analyze the imaging 105 
characteristics of GF-4 image and the moving ship wake characteristics in the image; In 106 
section 3, the methods proposed in this paper are described in detail, including image 107 
enhancement method, target detection method, data association method, etc. In section 108 
4, the experimental results and analysis of our method are explained. In section 5, we draw 109 


the conclusion. 110 
2. Analysis of ship wake characteristics in GF-4 Satellite optical remote sensing im- 111 
ages 112 


The spatial resolution of GF-4 image is 50m. A ship target with several hundred me- 113 
ters in length and tens of meters in width occupies only a few pixels in the GF-4 image. 114 
Moreover, many hulls may be coated with stealth materials and lack reflective features, 115 
making them difficult to detect directly. When the ship is moving, regular distribution of 116 
ripples will be generated in the tail, which is the wake of the ship, wake detection is an 117 
effective way to detect the ship, so this paper analyzes the characteristics of the wake to 118 
detect the ship target. 119 

The wake of a ship is mainly caused by the force between the hull (propeller) and the 120 
sea when the ship moves on the sea, which makes the sea water in the subsurface rise to 121 
the surface, thus forming the wake. Normally, the width of the wake is about 1 to 3 times 122 
the width of the ship, and the length of the wake is 1 to 20 times the length of the ship. 123 
The wake of a moving ship is mainly composed of kelvin wave, divergent wave, trans- 124 
verse wave, turbulent wave and breaking wave, as shown in Figure 1. The performance 125 
of wake in different detection methods is also different. For example, in SAR remote sens- 126 
ing images, kelvin wave, divergent wave, shear wave and broken wave can be observed. 127 
And kelvin wave, divergent wave, shear wave, turbulent wave and broken wave canbe 128 
observed in high resolution optical remote sensing images. In the GF-4 satellite optical 129 
remote sensing image, we can only see turbulent waves. The observed wake of turbulent 130 
wave appears as a spindle-shaped bright area in the GF-4 image, and its brightness and 131 
width decrease continuously along the opposite direction of the ship's motion. The main 132 
factors affecting the turbulence wake size in GF-4 images are ship size and sailing speed. 133 


Breaking Wave Region 


Kelvin Wave 
(19.5°) 
Motion Direction 


« i ae EA 
Moving Ship 
TE: Turbulent Wave 


Kelvin Wave Envelop Pa 
Transverse Waves 
Divergent Waves 
134 
Figure 1. Wake composition of moving ship. 135 


The coverage of GF-4 satellite remote sensing image is 500kmx500km, that is, 10,000 136 
pixels x 10,000 pixels. Figure 2(a) shows a real GF-4 image. In order to better analyze the 137 
characteristics of the GF-4 satellite optical remote sensing image and the ship wake, a 138 
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50kmx50km (1000pixelsx1000Pixels) area of interest is selected from the original image for 
analysis and illustration, which is shown in Figure 2(b). 


(a) (b) 


Figure 2. (a) Panchromatic remote sensing image of GF-4 satellite; (b) Enlarged view of the red area. 


The raw gray scale of GF-4 image is 16bit, and the theoretical maximum gray value 
is 65536. Figure 3 shows the comparison of Figure 2(b) before and after stretching. Figure 
3(a) is the two-dimensional(2D) view of the original GF-4 image, and Figure 3(b) is the 2D 
view of the stretched GF-4 image. Figure 3(c) is the three-dimensional(3D) view of Figure 
3(a). We can see that extremely bright bad pixels appear at (245,448) and dark bad pixels 
appear at (672,966). Therefore, it can be concluded that in GF-4 images, the maximum and 
minimum gray values are often bad pixels. In addition, the gray value difference between 
cloud and sea is not obvious, and the gray value change is mainly reflected in the floating 
change of cloud brightness, and the ship wake can hardly be found. Figure 3(b) is the 3D 
view of stretched GF-4 image. It can be seen that the gray difference between cloud and 
sea background is enlarged, the bad pixels at (245,448) and (672,966) are suppressed, and 
the ship wakes at (713,131) and (944,291) are highlighted. It is more conducive to the de- 
tection of ship wake later. 
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Figure 3. (a) 2D view of the original GF-4 satellite image; (b) 2D view of the stretched GF-4 satellite image; (c) 3D view of 
the original GF-4 satellite image (d) 3D view of the stretched GF-4 satellite image. 


GF-4 satellite optical remote sensing images often contain cloud clutter, ship wakes, 
and flaws. Figure 4 shows an enlarged view of the ship's wake, cloud clutter, and flaws in 
the area of Figure 2(b). After the analysis of the image, we can find that 1) The GF-4 satel- 
lite optical remote sensing image contains a lot of bright clouds, and the gray value of the 
cloud is higher than that of the ship's wake; 2) In the process of sea surface movement, the 
ship will form obvious wake, and the gray value of wake is higher than the sea back- 
ground; 3) Wakes of different ships have different sizes and brightness, which are related 
to the size of the ship itself and the speed of the ship; 4) Due to the very harsh electromag- 
netic radiation environment of the stationary orbit, some bad pixels appeared in the cam- 
era of the GF-4 satellite. There are several bright and black flaws in the remote sensing 
image, but the number of pixels of the flaws is less than ship wake; 5) The brightness of 
the cloud is variable, including large areas of bright cloud or local bright spot-shaped 
cloud. 
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(e) 


(a) © 


Figure 4. (a) stretched image of GF-4; (b) enlarged view of ship wake in red box areal (c) enlarged 
view of ship wake in red box area2; (d) enlarged view of flaw point in red box area3 (e) enlarged 
view of spot-shaped cloud in red box area4. 


3. The Method Proposed 


According to the above analysis, ship targets in GF-4 satellite optical remote sensing 
images are easily disturbed by clouds and flaws, and the brightness of ship wakes is lower 
than that of clouds and higher than that of sea background. Image stretching can better 
enhance ship wakes in images. The ship wake in the image is a turbulent wake with a 
certain shape, but it is still weak and occupies a small number of pixels in the image. At 
the same time, GF-4 satellite is a video-like satellite, which can continuously scan the ob- 
servation area and form image sequence. Therefore, the method of multi-frame associa- 
tion can be used to confirm the moving ship and remove false alarms. The target detection 
and tracking framework of sea-surface moving ships proposed in this paper is shown in 
Figure 5. 
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Figure 5. Proposed detection and tracking framework 


The method proposed in this paper includes four stages: image enhancement, sali- 
ency detection, shape check and multi-frame association. 

(1) Image enhancement stage: ANGS is used to enhance each frame of GF-4 image 
sequence to improve image contrast and highlight ship wakes in the image; 

(2) Target detection stage: Target detection based on visual saliency is the focus of 
this paper. MDDCM method is used to calculate the saliency map, and then the image is 
segmented according to the dynamic threshold value to obtain the location of the candi- 
date ship target. 

(3) Shape verification stage: Based on the detection results of MDDCM, the region 
where the target is located is binarized to obtain the shape of the ship wake and remove 
the false targets that do not meet the characteristics of the ship wake. 

(4) Multi-frame tracking stage: JPDA method is used for data association to confirm 
the real moving ship target from the candidate targets in multi-frame images. In addition, 
ship speed and its track are obtained, and false targets such as stationary islands are fur- 
ther removed. 


3.1. GF-4 satellite optical remote sensing image enhancement 


As analyzed above, we found that the pixels with the greatest brightness in GF-4 
satellite optical remote sensing image is often a flaw, and the brightness of the cloud is 
generally higher than that of the ship wake. Therefore, in order to effectively detect ships 
on the sea surface, it is necessary to enhance GF-4 image to enhance the contrast between 
ship wake and sea background. Generally, image enhancement methods are divided into 
spatial domain enhancement method and frequency domain enhancement method. Due 
to the uncertain size and shape of ship wake, frequency domain enhancement method is 
not suitable for image enhancement. Commonly used spatial domain image enhancement 
methods include histogram equalization, Laplace transform, Log transform, gamma 
transform and so on. In this paper, an adaptive nonlinear gray stretch (ANGS) method is 
used to suppress high-brightness clouds, and at the same time, to enhance the brightness 
of the ship wake. The formula of ANGS is as follows: 
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Where, G (x, y) is the original remote sensing image of GF-4, m is the mean value of 
the original image G (x, y), E is the stretch factor, which controls the slope of the stretch 
curve, eps is a very small value to prevent the formula from being meaningless when G (x, 
y) =0. E is an empirical value, and different stretching factor E has different stretching 
effects on the image. The larger the value of E is, the greater the gray contrast near the 
mean value m will be, and the gray value compression of high and low gray levels will 
also be stronger. 


3.2. MDDCM method for ship wake detection 


According to the previous analysis, the gray value of ship wake in GF-4 image is 
lower than that of clouds and land. Ship wake generally only occupies a dozen or even a 
few pixels in GF-4 image, which is a small and dim target. In recent years, algorithms 
based on HSV show good performance in dim and small targets detection. HVS algo- 
rithms generally use the local gray difference between the target and the surrounding 
background to calculate contrast, such as Local Contrast Measure (LCM) [25] and the im- 
proved algorithm based on LCM. Based on HVS theory, we propose a multi-scale dual- 
neighbor difference contrast measure (MDDCM) method for ship wake detection in GF-4 
images. 


3.2.1. DDCM window structure 


The traditional LCM window is shown in Figure 6(a), including 9 sub-blocks, namely 
the central block and the adjacent 8 sub-blocks. The traditional LCM algorithm measures 
the visual salience of the central block by calculating the contrast between the central block 
and the surrounding 8 sub-blocks. 
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Figure 6. (a) structure of LCM; (b) slide whole image by DDCM window; (c) structure of DDCM 


Different from the traditional LCM window, we designed a dual-neighborhood win- 
dow, which includes the center region, the middle region and the background region. By 
sliding the dual-neighborhood window on the whole image, the local contrast of the im- 
age is calculated and the saliency map is generated. The sliding process is shown in Figure 
6(b). The structure of the dual -neighborhood window is shown in Figure 6(c), which con- 
tains a central sub-block, 8 middle sub-blocks M1-M8 and 16 background sub-blocks B1- 
B16. 


3.2.2. DDCM local contrast calculation 
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From the previous analysis, we know that the ship's wake appears as a spindle- 
shaped bright area in the GF-4 image. When the wake is small, it appears as a gaussian 
spot, and when the wake is large, it appears as a long strip. The distribution of gaussian 
spot-like ship wake on the DDCM window is shown in Figure 7(a). Since the course of the 
hull is arbitrary, we use four angles of Figure 7(b), Figure 7 (c), Figure 7 (d) and Figure 7 
(e) to roughly describe the distribution of strip-like ship wake on the DDCM window 


(a) 


Figure 7. Distribution of ship wake in DDCM window 


Based on DDCM window structure, the difference between the three regions is used 
to measure the local contrast. The contrast between the central sub-block T and each sub- 


block M, in the middle region is represented by d(T,M,), and the expression of 
d(T,M,) is: 


d(T,M,)= 


ne —medny, if mean, —mean, > 0 ee m 


else 


Where, mean, represents the gray mean of the central T sub-block, mean,, repre- 


sents the gray mean of the middle sub-block M ,, and i=1,2...... 8. 
The contrast between the center sub-block T and the background region is repre- 
sented by D,, and the expression of D, is: 


, k=1,2,3...16 (3) 


B 


mean, —max(mean, ) if mean, —max(mean, ) > 0 
0 else 


Where, mean, represents the gray mean of the central sub-block T, mean, rep- 
resents the gray mean of the background sub-block B, , where, k =1,2...... 16, 
max(mean, ) represents the maximum value of mean, in 16 background subblocks . 


Generally, the gray value of the central region where the ship wake is located is higher 
than that of the outer background region, so D, can effectively suppress cloud clutter 
and highlight the ship wake. 

Ship wakes are generally distributed symmetrically along the central axis. When the 
central sub-block of the DDCM window passes through the center of ship wake, part of 
ship wakes may be distributed in the middle region. Therefore, the contrast between the 
central region T and the middle region M can be expressed by D, , and the expression of 


n 


D isas follows: 
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d(T,M,)xd(T,M,) if max(mean, )=mean, or max(mean, )=meany, 

d(T,M,)xd(T,M,) elseif max(meany,)=meany,, or max(mean,, ) = meany, 
"Vd (T,M,)xd(T,M,) else if max(mean,,)=mean,, or max(mean,, )=meany, “ 

d(T,M,)xd(T,M,) elseif max(meany, )=meany,, or max(mean, )=mean,, 


Where, max(mean,, ) represents the maximum value of mean,, for 8 sub-blocks 


in the middle region, and the sub-block corresponding to max(mean,, )may be the sub- 


block where the ship wake symmetry axis is located. So, we use the sub-block perpendic- 
ular to the sub-block of max(mean,, ) to calculate the contrast between the central re- 
gion T and the middle region M. 

Figure 8(a) shows the GF-4 satellite optical remote sensing image, which contains a 
ship wake with an angle of about 45°. When the DDCM window slides to the center of the 
ship wake, the mean value of T in the central region reaches the maximum value. Among 
~ the 8 sub-blocks mean, in the middle region, meany, has the largest gray mean, so 


M, and M, perpendicular to M, are used to calculate D, , that is, 


m 


D, =d(1,M,)xd(T,M,). 


a 
‘ea 
WJ) 


(a) 
Figure 8. (a)A ship wake in the GF-4 satellite optical remote sensing image (b)Ship wake covered by DDCM window. 


If the size of the sub-block T in the central region of the DDCM window is k x k, the 
visual saliency of the k-size DDCM can be expressed by DDCM,, and the formula for 


DDCM, is as follows: 
DDCM, = D, xD, (5) 


Where, D,,, is the contrast between the central region and the middle region, and 


D, is the contrast between the central region and the background region. By setting the 


value of DDCM window size k and using DDCM window to slide through the whole GF- 
4 image, we can get a single-scale DDCM saliency map. 
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3.2.3. MDDCM local contrast calculation 295 


Since the size of ship wake is variable, it is influenced by many factors such as hull 296 
size, sailing speed, image spatial resolution, etc. A single-scale DDCM window cannot be 297 
suitable for detecting ship wakes of all sizes. Therefore, we use the Multi-scale Dual- 298 
neighbor Difference Contrast Measure (MDDCM) method to detect ship wakes. 299 


The DDCM saliency map of size k can be represented as DDCM,, and the saliency 300 
of pixel P(i, j) in the image canbe represented as DDCM, (i, j) The expression of the 301 


saliency MDDCM, (i, j) of pixel P(i, j) in the MDDCM algorithm is as follows: 302 
MDDCM, (i, 7)=max(DDCM, (i, j)), k =2,3,4,5...N (6) 

Where, k is the size of DDCM, the minimum value is 2 and the maximum value is N. 303 

Therefore, the MDDCM with the largest scale of K can be expressed as: 304 
MDDCM=max(DDCM,), k = 2,3,4,5...K (7) 

3.2.3. Adaptive extraction of ship position based on MDDCM 305 


After MDDCM calculation, the MDDCM saliency map of the image can be obtained. 306 
The contrast of the ship wake area has been enhanced, while bright flaws, random noise, 307 
cloud clutter and sea background clutter have been effectively suppressed, and the signal- 308 
to-noise ratio(SNR) has been significantly improved. In order to extract the target, we nor- 309 
malize the saliency map to make it distributed between 0 and 1, and then use an adaptive 310 
threshold to segment the saliency map to obtain the location of the target. The expression 311 
of threshold t used for saliency map segmentation in this paper is: 312 


T = U+ Ap (8) 


Where, u is the mean value of MDDCM saliency map, o is the standard deviation of 313 
MDDCM saliency map. 4,, is the segmentation factor, usually ranging from 20 to 50. Us- 314 


ing the threshold t to segment the saliency map, we can extract the position of the target, 315 
which is the candidate ship. 316 


3.3. Shape-based false alarm removal 317 


The detection result of MDDCM saliency map can only get the position of candidate 318 
ships in the image, but it cannot show the size and shape of candidate ship. In fact, the 319 
extraction result in MDDCM saliency map is usually the center position of ship wake. 320 
Therefore, it is necessary to segment the target candidate region to obtain the contour in- 321 
formation of the candidate ship after obtaining the location of the ship using MDDCM, 322 
and then we can use the shape verification to remove false targets such as spot-shaped 323 
cloud. 324 

Based on the location of candidate target extracted by MDDCM, we extract 64 pixels 325 
x64 pixels regions of interest (ROI), and then use Otsu method to perform binary segmen- 326 
tation of ROI to separate target from sea surface background. The formula is as follows: 327 


1, S(x, y)2T 


S ; = 
(%y) hee y)<T O) 


Where, T is the segmentation threshold of Otsu method. For the segmented image, 328 
the connected region centered on the detection result of MDDCM saliency map is the ship 329 
region. The detection results of MDDCM saliency map often contains false alarms. 330 
Through binary segmentation of the target area, the target contour can be obtained, and 331 
then the target shape characteristics can be verified. 332 
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The spatial resolution of GF-4 satellite optical remote sensing image is 50m, generally 
the width of ships is within 50m, AINA A ccording to the 
above analysis, the wake width of the ship is about 1 to 3 times of the hull width, and the 
wake length is 1 to 20 times of the hull length. Therefore, the width of ship wake in GF-4 
image is only a few pixels, while the length may reach dozens of pixels. Generally, the 
larger the wake width, the larger the size of the ship or the faster the motion speed, which 
means that the turbulent wake formed behind the ship will be longer, so there is a positive 
correlation between the length and width of the ship wake. Based on this feature, we can 
verify the shape of targets and remove the false alarm targets that do not meet the shape 
characteristics of the wake. If the length of the smallest enclosing ellipse of the target area 


is Length, , and the width is Width, . The length-to-width ratio is represented by R, . 
The width Width, and the length-to-width ratio R 


certain range. We use the following formula to verify the shape of the ship's wake: 


of the ship's wake should meet a 


lw 


Width, „„ < Width, < Width „ax (10) 
Re Width, +1 an 
2 
Lenet 
_ Length, SR (12) 


Tw Width, th 


Where, Width. and Width are the minimum and maximum value ranges of 


min max 
wake widths. Limiting the range of Width, can effectively remove small noise points, 


large clouds and islands, etc. R, is the adaptive threshold value associated with 


th 
Width,. Using R,, to verify the length-width ratio R, 
characteristics of ship wake. 


can better reflect the physical 


w 


3.5. Ship tracking based on JDPA 


The GF-4 optical satellite is equipped with a staring camera. The shortest single-spec- 
trum repetitive imaging time is only 5s, and the shortest full-spectrum repetitive imaging 
time is 100s. It has video-like satellite characteristics and can meet the needs of tracking 
ship targets moving at high speeds at sea. [11]. Using GEO satellites to detect and track 
marine moving ships can obtain its moving track and estimate its movement trend. At the 
same time, it can further remove false alarm targets such as stationary islands, irregular 
clouds, and random noise by data association. 

JPDA is a classic data association method. Its main idea is to comprehensively con- 
sider all targets and measurements. According to the association between all measure- 
ments and all targets. JPDA first establishes a confirmation matrix, and then lists all pos- 
sible events based on the criterion that the target has at most one measurement infor- 
mation in a period, and a measurement has and only one source. Finally, all events are 
connected in probability to obtain the final status update value of each target. 

In JPDA method, the predicted state equation of the target is defined as: 


X'(k|k-1) = F'(k-))X'(k-1]k-1) (13) 


Where, X'(k |k—1) represents the state vector of t at time k predicted by time 


k—1,and F'(k—-1) is the state transition matrix of t attime k—1. 
The prediction vector of measurement target is defined as: 
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Z'(k|k—-1) = H(k)X'(k|k—-1) (14) 
Where, H(k) represents the measurement target matrix. 369 
According to the Kalman filter formula, the equation for calculating the status update 370 
is as follows: 371 
X'(k|k) =X'(k|k-1) 4 K' (KV! (k) (15) 
Where, K'(k) represents the Kalman gain matrix of t at time k, and V'(k) 372 
represents the combined information of t attime k. 373 
Define the associated area as: 374 
A T — A 
A(k)=|z(k)-2(k|k-1)| S” (k)|z(k)-2(k|k-1)|<7 (16) 
Where, S~ (k) represents the innovation covariance matrix at time k, and yisa 375 
fixed threshold, which can be obtained from the % : distribution table. 376 
As an extension and optimization of the PDA method, the JPDA method mainly in- 377 
troduces the concept of joint events within the association cycle: 378 
M, 
TOAG a7 
= 


Where, 0, (k ) is the joint associated event, k is the time, @, (k ) is the i thjoint 379 


event at the time k,M, represents the total number of measured targets attime k,and 380 

i 
A 
at time kin the joint event of i. When t=0Q, it means that the measured target j has 382 


(k) represents the event that whether the measured target j is related to track £ 381 


no related events, that means a false alarm. 383 

JPDA method assumes that the states of different targets are independent of each 384 
other under historical measurement conditions. The state estimation can be converted into 385 
irrelevant estimation of different targets, and edge association probability needs to be cal- 386 
culated based on joint probability, that is, to calculate the sum of the associated events 387 
related to the target. 388 

We use the target location extracted by the MDDCM method as the data source for 389 
data association (shape verification has been passed). Then, JDPA is used to associate the 390 
target data in multi-frame GF-4 image to obtain the ship target's track information and 391 
calculate the average speed. The candidate targets in GF-4 satellite image sequences often 392 
contain false targets. Therefore, we use constraints such as the minimum associated 393 
frames, the maximum target speed, the minimum target speed and the minimum moving 394 
distance to further remove false targets and filter the displacement of the stationary tar- 395 
gets caused by the camera shaking. 396 


4. Experiment and Analysis 397 


Five groups of GF-4 satellite optical remote sensing image sequences are used for 398 
experiments, and the experimental results are analyzed. The effectiveness and perfor- 399 
mance of the proposed method are demonstrated by the results verification and compar- 400 
ison with other methods. 401 


4.1. Experimental data 402 

The GF-4 satellite has imaging capabilities in the visible near-infrared spectrum and 403 
mid-wave infrared spectrum. There are 6 imaging bands B1-Bé6 in the visible near-infrared 404 
spectrum, of which the B1 band is the panchromatic band. The wake of moving ship on 405 
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the sea surface is more significant in the B1 panchromatic band. We use the panchromatic 
optical remote sensing images of GF-4 satellite as experimental data. The width of the GF- 
4 image is 500kmx500km, which is 10000 pixelsx10000 pixels. The ship's wake generally 
only occupies a few pixels in the remote sensing image. In order to better observe the 
ship's wake and verify the experimental results, we cut out an area of 25km x 25km (500 
pixels x 500 pixels) from the original image as the experimental area. In order to fully 
verify the effectiveness of the proposed method, 5 groups of GF-4 satellite optical remote 
sensing panchromatic image sequences are used as experimental data, and the experi- 
mental data included various scenes, such as few clouds, thick clouds, speckled clouds, 
islands, and multiple targets. Moreover, the size of the ship's wake is also different in these 
experimental data, in order to better verify whether the method can support target detec- 
tion of different scales. The description of the experimental data is shown in Table 1: 


Table 1. Data used in the experiment 


Image Sequence Band Number of Images Features of the scene 

Image Sequence1 B1(panchromatic) 28 Multi ship 

Image Sequence2 B1(panchromatic) 28 Having island 

Image Sequence3 B1(panchromatic) 28 Dense cloud 

nage eguen B1(panchromatic) DA spot-shaped cloud + Thick 
clouds 

Image Sequence5 B1(panchromatic) 12 Thick clouds 


4.2. Detection and comparison 
3.1.1. Detection 


According to our detection and tracking framework, ANGS should be used to stretch 
single frame image to improve image contrast and highlight ship target before ship target 
detection of single frame remote sensing image. According to the stretching experiment 
of several groups of GF-4 satellite panchromatic optical remote sensing images, it is found 
that the stretching effect is better when the value of E is set between 6 and 8. We set pa- 
rameter E to 6 in the experiment. MDDCM algorithm is used for multi-scale target detec- 
tion based on the enhanced image. The range of MDDCM window size depends on image 
resolution and target size. According to the previous analysis, the width of the ship's wake 
of the GF-4 image is about a few pixels, and the length may reach dozens of pixels. There- 
fore, the size k of the MDDCM window is set in the range of 3~6 for multi-scale saliency 
map calculation. Then, the binary segmentation of MDDCM saliency map needs to deter- 
mine the segmentation threshold t between the target and the background. The segmen- 


tation factor Åy, of t is an empirical value, and we set /,,as 20. The parameter descrip- 


th 


tion is shown in Table 2: 


Table 2. Parameter values in ship detection experiments 


Method Symbol Parameter description Value 
ANGS E stretch factor 6 
Kamin min size of center window 2 
MDDCM Kmax max size of center window 4 
Åh segmentation factor 20 


According to the parameter values in Table 2, we use ANGS method and MDDCM 
method respectively to carry out image enhancement and saliency map calculation for 
five groups of GF-4 image sequences, and then detect candidate ship targets by binariza- 
tion segmentation. The experimental results are shown in Figure 9: 
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Figure 9. (a) original image; (b) 3D view of original image; (c) stretched image by ANGS; (d) 3D view of stretched image 
by ANGS; (e) saliency map by MDDCM,; (f) 3D view of saliency map by MDDCM; (g) Ship wake detection results. 


By comparing Figure 9(b) and Figure 9(d), it can be found that the ANGS method can 
effectively suppress bright clouds and islands, improve image contrast and visual effects, 
and highlight ship targets. According to Figure 9(e) and Figure 9(f), it can be found that 
MDDCM method can significantly suppress background and improve the SNR of the im- 
age. Figure 9(g) is the detection result after the binary segmentation of the MDDCM sali- 
ency map. It can be seen that: The first group of experimental data contains 6 ship targets, 
and the ship passes through the cloud, our method can detect all 6 ship targets; The sec- 
ond group of experimental data contains islands and ship targets. The brightness of is- 
lands is much higher than that of ship targets. Our method can detect ship targets and 
suppress high-brightness islands; The third group of experimental data contains two ship 
targets that can be detected; The lower right corner of the fourth group of experimental 
data contains a ship target with significant wake, which can be accurately detected by our 
method. At the same time, due to the influence of spot-shaped cloud, false alarms appear, 
which can be further removed by the following methods in this paper; The fifth experi- 
ment included two ship targets with significant wakes, both of which could be detected. 
3.1.1. Comparison 

We have carried out comparative experiments between MDDCM Method and other 
visual saliency methods, including LCM method [25] and Multiscale patch-based Contrast 
Measure (MPCM) method [26]. Local Peak signal-to-noise Ratio (PSNR) method [21], and 
Spectral Residual (SR) method based on frequency domain [27]. Experimental parameter 
settings of MDDCM are shown in Table2, and experimental parameter settings of other 
methods are shown in Table 3: 


Table 3. experimental parameter values of other visual saliency methods 


Method Symbol Parameter description Value 
K size of center window 3 
LEM An segmentation factor 5 

K size of center window 3, 5, 7 

hen Ai segmentation factor 20 
Kin size of inner window 10 
PSNR Kout size of outer window 20 
th fixed segmentation threshold 4 
n size of convolution kernel 3 
a An segmentation factor 20 


Using the five groups of experimental data introduced above, the comparative ex- 
erimental results of ship detection are shown in Figure 10: 
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b) 3D view of tease map by LCM 


c) ship detection result of saliency map by LCM 
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(h) 3D view of hens map by SR 
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(i) ship detection result of saliency map by SR 


(j) 3D view of saliency map by MDDCM 


(k) ship detection result of saliency map by MDDCM 


Figure 10. (a) Stretched image by ANGS; (b) 3D view of saliency map by LCM; (c) ship detection result of saliency map 
by LCM; (d) 3D view of saliency map by MPCM; (e) ship detection result of saliency map by MPCM; (f) 3D view of sali- 
ency map by PSNR; (g) ship detection result of saliency map by PSNR; (h) 3D view of saliency map by SR; (i) ship detec- 
tion result of saliency map by SR; (j) 3D view of saliency map by MDDCM; (k) ship detection result of saliency map by 
MDDCM. 


From Figure 10, we can see that: LCM algorithm is prone to the interference of clouds 
and islands. In the second group of data, it missed the detection of ships, while in the 
fourth group of data, it missed the detection of ship targets with obvious wakes, and a 
large number of false alarms appeared in the fifth group of data. This is because the LCM 
algorithm uses the maximum gray value of the center window instead of the average gray 
value to participate in the calculation, and the window size is fixed. Moreover, it should 
be noted that the segmentation factor 4,, of the LCM saliency map is set to 5, because 
the contrast of the LCM saliency map is not as good as other visual saliency methods. If 
the default value of 4,, is 20, it is almost impossible to segment and detect the target. 


MPCM has a good detection performance, but there are some false alarms appear in the 
first and fourth group of data. At the same time, MPCM algorithm has a very good back- 
ground suppression ability, which can improve the SNR of images. The background sup- 
pression ability of the PSNR method provided by the paper [21] is not good, resulting in 
the low SNR of the saliency map. Some targets are missed in the first group of data, and 
a large number of false alarms appeared in the fourth group of data, indicating that the 
anti-interference ability of PSNR is insufficient. This may be due to the segmentation 
threshold provided in literature [21]. Fixed threshold will make it difficult to balance the 
relationship between missed detection and false alarm. SR method also has good detection 
performance, but its background suppression ability is mediocre. It missed some targets 
in the first group of data and failed to detect targets in the second group of data, which 
may be related to the value of the segmentation factor (set as 20). Its detection effect in the 
third and fourth group of data is mediocre. In contrast, our MDDCM method successfully 
detected all targets with a low false alarm rate and excellent background suppression 
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ability, which can greatly improve SNR and has the best detection performance among all 
comparison methods. Although there are false alarms in the second and fourth group of 
data, these false alarms can be removed by our follow-up method. 


4.3. Shape-based false alarm removal 

According to the previous analysis of the image resolution, as well as the size and 
shape of the ship's wake. The value of Width is set to 2, the value of Width 
to6,and R,, is the adaptive threshold, which is determined by Width. The purpose of 


shape verification is to further remove false alarms from candidate ship targets, especially 
to avoid the interference of spot-shaped cloud to target detection. Table 4 shows the shape 
verification results of some candidate ship targets. 


is set 


max 


Table 4. Shape-based verification 


Candidate Target Width Length Rw Rin Verification 
2.6277 11.0888 4.2200 1.8138 True 
3.1409 8.6349 2.7492 2.0704 True 
3.2342 10.2844 3.1799 2.1171 True 
4.4772 7.1736 1.6022 2.7386 False 
3.6485 12.4416 3.4100 2.3243 True 
3.0827 4.9899 1.6187 2.0414 False 
6.2077 10.1860 1.6409 3.6038 False 
3.8072 8.6802 2.1068 2.4036 False 
5.0590 7.0992 1.4033 3.0295 False 
3.0048 13.3384 4.4391 2.0024 True 


The candidate ship targets detected by MDDCM algorithm often contain false alarms, 
such as small islands and spot-shaped cloud, which tend to have high brightness. Figure 
11(a) is the detection result of MDDCM algorithm, where target C is an island and target 
A is a moving ship. The R, of target C does not meet the threshold R, , so itis removed 


from the candidate target, and the result is shown in Figure 11(b). 
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(a) detection result of MDDCM (b) false alarm removal by shape verification 


Figure 11. (a) detection result of MDDCM,; (b) false alarm removal by shape verification. 


4.4. Ship Tracking 


After the target detection and shape verification of single frame image in the image 
sequence, the candidate ship of single frame image can be obtained. Taking the center 
pixel of each candidate target as the data source, the multi-frame data association of the 
image sequence can be carried out to obtain the track and speed of the ship. At the same 
time, conditions such as the minimum number of associated frames, the maximum speed, 
the minimum speed, and the minimum moving distance can be used to further remove 
false targets. The corresponding parameter settings are shown in Table 5: 


Table 5. Constraint parameters for ship tracking 


Item value 
Minimum association frames 3 
Minimum speed 10 km/h 
Maximum speed 80 km/h 
Minimum distance 2km 


Figure 12 shows the process of associating with three frames of data by JPDA method. 
The previous method detects two targets Al and B1 in Figure 15(a), A2 and B2 in Figure 
12(b), and A3 and B3 in Figure 12(c). The track and speed of target A are obtained success- 
fully by associating three frames of data. However, the false alarm target B is a fixed island, 
which is not associated with a stable track, so it is filtered out. 
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Figure 12. (a) frame 1 with ae A1 and B1 (b) frame 2 with target A2 and B2 (c) frame 3 with target A3 and B3 (d) data 


JPDA algorithm is used for data association of candidate targets detected by five 
groups of experimental data, and the obtained track information is shown in Table 6: 


Table 6. Tracking result 


Image Sequence Total frames Track Id association frames Mean Speed 
Track] 28 48.75 km/h 
Track2 28 41.93 km/h 
Track3 22 47.06 km/h 
Image Sequencel 28 Track4 12 52.74 km/h 
Track5 8 40.82 km/h 
Track6 6 41.26 km/h 
Track7 6 40.83 km/h 
Image Sequence2 28 Track] 28 37.27 km/h 
imape Sedie 28 Track1 28 48.24 km/h 
Track2 15 43.51 km/h 
Image Sequence4 12 Track1 9 38.72 km/h 
Image Sequences 24 Track1 6 18.78 km/h 
Track2 12 56.42 km/h 


4.6. Results evaluation 


In order to verify the effectiveness and performance of the proposed method, we use 
Recall, Precision, and F-Score to evaluate the method and compare it with other methods. 
Recall represents the effectiveness of the detection, Precision represents the accuracy of 
the detection, and F-Score is the comprehensive response of Recall and Precision. The cal- 
culation formulas of Recall, Precision and F-Score are as follows: 


Recall = OE (18) 
TP + Ne 
Precision = TP (19) 
Ny + Npp 


F-Score = 2x Rcall x Precision 


SE (20) 
Rcall+Precision 


Where, N,, and N py are the number of correctly detected targets and the number 


of ships that have not been detected respectively. Npp is the number of false targets de- 
tected. According to the previous experimental data and experimental parameters, the 
detection results of the five groups of experimental data are statistically analyzed, and the 
comparison between the proposed MDDCM method and other visual saliency methods 
is shown in Table 7: 


Table 7. Performance comparison between MDDCM and other methods 


Method Nret+tNen  Nerre+Nere Np NEN Nre Recall (%) Precision (%) F-Score (%) 
LCM 284 177 51 97 81 62.3 70.4 
MPCM 257 198 20 59 90 77 83 
PSNR 218 223 181 37 42 83 81.2 81.1 
SR 207 175 43 32 80 84.5 82.2 
MDDCM 252 213 5 39 98 84.5 90.75 


532 


533 


534 
535 
536 
537 
538 


539 


540 
541 
542 
543 
544 


545 


Sensors 2021, 21, x FOR PEER REVIEW 22 of 23 


The experimental results show that LCM has the worst performance among all meth- 546 
ods, and the proposed MDDCM method has the best performance. MPCM method and 547 
MDDCM method in this paper have higher Recall, but MPCM method has more false 548 
targets. Although SR method has a higher Precision, it has a lower Recall rate. 549 

Although the MDDCM method has the highest Recall among all methods, reaching 550 
98%, its Precision is only 84.5%, and there are still many false targets. The proposed shape 551 
verification method can further remove the false targets that do not meet the characteris- 552 
tics of ship wake. Then, JPDA is used for multi-frame data association of candidate targets 553 
in a single frame image, which can further remove the false targets that cannot be associ- 554 
ated with stable track. Thus, the Precision of detection can be improved. Table 8 shows 555 
the improvement of detection performance after shape verification and multi-frame data 556 


association. 557 
Table 8. Performance improvement by shape verification and data association 558 
Method Recall (%) Precision (%) F-Score (%) 
MDDCM 98 84.5 90.75 
After shape verification 96 94.2 95.1 
After data association 95 99 98 


Experimental results show that after shape verification, some false targets are further 559 
filtered, and the Precision is increased from 84.5% to 94.2%. After multi-frame data asso- 560 
ciation, the Precision is improved to 99% and most false targets are removed. Some real 561 
targets are filtered out due to shape verification, while some targets are discarded because 562 
they are not associated with a stable track, resulting in a slight decrease in recall rate. The 563 
comprehensive evaluation F-Score increased from 90.75% to 98%. 564 


5. Conclusions 565 


Geostationary optical remote sensing satellites can continuously observe a fixed area, 566 
GF-4 satellite has the advantages of high temporal resolution and large coverage, which 567 
make it possible to continuously observe ships moving on the sea surface. However, due 568 
to the spatial resolution of the image, as well as the interference of high-brightness clouds, 569 
islands, sea clutter, etc., the ship target is weak and small in the image, which brings great 570 
difficulties to the direct detection of the ship. Based on the wake characteristics of moving 571 
ships on the sea surface, a moving ship detection and tracking method for GEO satellite 572 
images is proposed in this paper. The method includes four steps: image enhancement, 573 
target extraction, removing false alarms using shape features, and multi-frame association 574 
tracking. Firstly, the ship wake in the image is enhanced by ANGS, and then the candidate 575 
ship target is extracted by MDDCM algorithm based on the visual salience theory. Then, 576 
the false target is further filtered by the shape feature of the ship wake. Finally, the track 577 
information of moving ship target is obtained by JPDA multi-frame data association. 578 

Through experimental analysis and comparison with other methods, the proposed 579 
method has good detection performance, especially in the case of weak targets or complex 580 
clouds environment, and can effectively detect and track the moving ships on the sea sur- 581 
face in the GEO optical remote sensing satellite images. Since we only use GF-4 satellite 582 
panchromatic image data for experiment, sea surface moving target detection and track- 583 
ing based on GF-4 satellite multi-spectral image data can be further studied in the future. 584 
The method proposed in this paper can also be applied to other moving target monitoring 585 
applications of medium and low resolution optical remote sensing satellites. 586 

587 
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